{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 北京大学开源的LAC模型\n",
    "\n",
    "> https://github.com/lancopku/pkuseg-python\n",
    "\n",
    "> [PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation](https://arxiv.org/pdf/1906.11455)\n",
    "\n",
    "## Install\n",
    "\n",
    "Ubuntu 24 上在 virtual env 内使用pip直接安装会有问题，需要下载下载源码包，然后手动安装。\n",
    "\n",
    "```bash\n",
    "python setup.py build_ext -i\n",
    "```\n",
    "\n",
    "- build_ext 表示构建扩展模块，有的代码是cython的。\n",
    "- -i 表示 inplace，表示把生成的文件放在当前目录下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['pkuseg-python', '主要', '基于', '经典', '的', 'CRF', '模型', '，', '辅以', '我们', '提出', '的', 'ADF', '训练', '方法']\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "\n",
    "# 下面这个目录是源码目录（需要执行上面的构建命令）\n",
    "sys.path.append(os.path.expanduser(\"~/packages/git/pkuseg-python\"))\n",
    "\n",
    "import pkuseg\n",
    "\n",
    "seg = pkuseg.pkuseg(model_name='default')\n",
    "txt = seg.cut('pkuseg-python主要基于经典的CRF模型，辅以我们提出的ADF训练方法')\n",
    "print(txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FAQ\n",
    "\n",
    "###  运行错误 ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject.\n",
    "\n",
    "构建时使用的 numpy 版本为 1.x ，而 June 16th 2024 发布的 numpy 2.0.0 开始不再兼容1.x，需要在当前python环境中降级numpy到1.x版本。使用jupyter的情况下，降级numpy之后记得重新加载kernel。\n",
    "\n",
    "### 模型文件下载到哪了\n",
    "\n",
    "日志里面显示：\n",
    "Downloading: \"https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/medicine.zip\" to /home/gdme1320/.pkuseg/medicine.zip\n",
    "\n",
    "### 下载的模型文件应该放在哪里、如何使用\n",
    "\n",
    "`~/.pkuseg` 是模型的下载缓存目录。\n",
    "在源码 __init__.py 中有如下代码：\n",
    "```python\n",
    "        if model_name in [\"default\"]:\n",
    "            config.modelDir = os.path.join(\n",
    "                os.path.dirname(os.path.realpath(__file__)),\n",
    "                \"models\",\n",
    "                model_name,\n",
    "            )\n",
    "        elif model_name in config.available_models:\n",
    "            config.modelDir = os.path.join(\n",
    "                config.pkuseg_home,\n",
    "                model_name,\n",
    "            )\n",
    "            download_model(config.model_urls[model_name], config.pkuseg_home, config.model_hash[model_name])\n",
    "        else:\n",
    "            config.modelDir = model_name\n",
    "```\n",
    "所以如果是default模型，则需要放在源码目录下的models/default目录下。\n",
    "其他模型，需要放在配置的 pkuseg_home 目录下。\n",
    "\n",
    "从github上下载的 default_v2 模型，是没有在源码中声明的：\n",
    "config.py中:\n",
    "```python\n",
    "    model_urls = {\n",
    "        \"postag\": \"https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/postag.zip\",\n",
    "        \"medicine\": \"https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/medicine.zip\",\n",
    "        \"tourism\": \"https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/tourism.zip\",\n",
    "        \"news\": \"https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/news.zip\",\n",
    "        \"web\": \"https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/web.zip\",\n",
    "    }\n",
    "    model_hash = {\n",
    "        \"postag\": \"afdf15f4e39bc47a39be4c37e3761b0c8f6ad1783f3cd3aff52984aebc0a1da9\",\n",
    "        \"medicine\": \"773d655713acd27dd1ea9f97d91349cc1b6aa2fc5b158cd742dc924e6f239dfc\",\n",
    "        \"tourism\": \"1c84a0366fe6fda73eda93e2f31fd399923b2f5df2818603f426a200b05cbce9\",\n",
    "        \"news\": \"18188b68e76b06fc437ec91edf8883a537fe25fa606641534f6f004d2f9a2e42\",\n",
    "        \"web\": \"4867f5817f187246889f4db259298c3fcee07c0b03a2d09444155b28c366579e\",\n",
    "    }\n",
    "    available_models = [\"default\", \"medicine\", \"tourism\", \"web\", \"news\"]\n",
    "    models_with_dict = [\"medicine\", \"tourism\"]\n",
    "```\n",
    "这里源码和文档已经不一致了，要么把default_v2改成default，要么改源码。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
